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Summary 

Over  the  past  decade,  Virtual  Environment  (VE)-based  training  systems  have  become  commonplace 
within  the  military  training  domain.  These  systems  offer  such  benefits  as  small  footprint,  rapid 
reconfiguration,  and  enhanced  training  delivery.  In  addition,  they  appear  to  offer  significant  relief  for 
a  market  starved  for  low  cost  training  systems,  and  hold  great  potential  as  effective  training  tools. 
Yet,  all  too  often  the  human  element  is  taken  for  granted,  with  systems  being  designed  to  incorporate 
the  latest  technological  advances,  rather  than  focusing  on  enhancing  the  user's  experience  within  the 
VE  -both  from  a  training  and  human  factors  perspective.  It  is  precisely  this  shift  in  design  philosophy, 
from  techno  centric  to  human  centric  that  represents  the  next,  greatest,  challenge  to  developing 
effective  VE-based  training  systems. 

Interaction  with  VE  involves  the  ability  of  individuals  to  effectively  perform  essential  perceptual- 
sensory-motor  tasks  within  the  virtual  world.  More  specifically,  this  can  involve  the  ability  to  move 
about  the  VE,  manipulate  virtual  objects,  locate  virtual  sounds,  deal  appropriately  with  physical 
constraints,  or  perform  visual  tasks  (i.e.,  discriminate  colors;  judge  distance;  search  for,  recognize, 
and  estimate  the  size  of  objects).  Interactive  technologies  include  multi-modal  3D  displays  and  input 
devices,  real-time  rendering,  and  distributed  simulation  (i.e.,  multiple  user  interaction  through 
networked  VE  systems).  These  technologies  define  how  the  environment  is  portrayed  and  how  it 
responds  to  user  actions.  The  design,  synthesis,  and  analysis  of  new  interaction  technologies  will  be 
based  on  our  growing  understanding  of  human  perception  and  action  in  VEs.  Tools  are  needed  to 
provide  a  more  comprehensive  assessment  of  the  quality  of  interaction. 

The  Office  of  Naval  Research's  Virtual  Technologies  and  Environments  (VIRTE)  Program  was 
developed  to  address  this  design  challenge.  VIRTE  is  focusing  on  developing  two  capabilities  central 
to  applying  VE  technology  to  training:  (1)  improving  the  quality  of  interaction  provided  by  VE,  and 
(2)  applying  advanced  training  aids  and  methodologies  to  real  Navy  and  Marine  Corps  requirements. 
Quality  interaction  is  essential  for  making  VE  usable  for  extended  training  and  for  applying  VE 
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technology  to  build  combat  simulators.  Advanced  training  aids  and  methodologies  are  being 
developed  to  derive  full  benefit  from  the  flexible  new  medium  of  VE. 

Our  approach  takes  maximum  advantage  of  the  growing  knowledge  base  in  human-centric  design  to 
build  systems  that  are  well-suited  for  enhancing  performance  of  real-world  tasks.  In  addition  to 
traditional  Knowledge  Acquisition  and  Knowledge  Engineering  approaches,  our  interdisciplinary 
development  team  is  utilizing  team  task  analyses,  motion  sickness  evaluations,  and  usability  analyses. 
The  team  has  also  developed  a  novel  methodology  for  evaluating  team-based  performance 
enhancement  in  VE  systems.  This  cross-disciplinary  approach  gives  the  team  a  unique  perspective 
into  designing  effective,  low  cost,  deployable  simulations.  The  technology  testbeds  that  are  being 
developed  are  useful  not  only  for  stand-alone  training,  but  also  for  embedded  training,  team  training, 
and  mission  rehearsal. 


Introduction 

Virtual  Environment  training  Systems 

Throughout  the  past  decade,  the  United  States  military  training  community  has  turned  its  focus  to 
Virtual  Environment  (VE)  technologies  as  a  means  for  providing  rapidly  deployable,  easily 
configurable,  affordable  training  solutions.  As  a  result,  a  veritable  barrage  of  (VE)-based  training 
systems  have  been  developed  as  potential  solutions  to  a  range  of  training  requirements.  While  these 
systems  typically  offer  the  latest,  cutting  edge  visual  and  hardware  technologies,  all  too  often  the 
human  element  is  taken  for  granted  (Cohn,  Breaux,  Nguyen  Schmorrow,  2001).  A  direct  result  of  this 
techno-centric  design  philosophy  is  that  the  effectiveness  of  these  systems  is  all  too-rarely  objectively 
evaluated.  Consequently,  the  likelihood  of  designing  a  final  product  with  a  strong  ‘gee  whiz’  factor, 
combined  with,  at  best,  an  undefined  ‘training  effectiveness’  quotient,  is  quite  high,  leading  users  to 
over-estimate  the  value  of  their  training.  In  the  operational  setting,  these  over-estimations  can  have 
costly  -if  not  tragic-  consequences. 

Human-centric  Design  Principles 

Yet  this  need  not  be  so.  In  parallel  to  the  quantum  leaps  made  in  the  technology  realm,  human  factors 
specialists  have  made  significant  strides  in  developing  human-centric  design  methodologies  for 
ensuring  that  training  systems  satisfy  the  needs  of  the  users  for  whom  they  are  intended.  The  crucial 
element  that  has  been  missing  is  an  underlying  philosophy  linking  these  techniques  in  a  logical 
fashion. 

Real  World  Example 

Virtual  Technologies  and  Environments  (VIRTE) 

The  positive  impact  resulting  from  implementing  these  principles  is  best  illustrated  through  an 
example  of  an  actual  VE  training  program,  the  Office  of  Naval  Research’s  Virtual  Technologies  and 
Environments  (VIRTE)  Program.  VIRTE  spans  several  levels  of  simulation  and  will  ultimately 
provide  a  networked,  deployable  system  for  training  a  myriad  of  concepts  relating  to  a  specific  type 
of  warfare  known  as  Expeditionary  Warfare.  Expeditionary  Maneuver  Warfare  (EMW)  ...  “is  the 
union  of  our  core  competencies;  maneuver  warfare  philosophy;  expeditionary  heritage;  and  the 
concepts  by  which  we  organize,  deploy,  and  employ  forces”  (Department  of  the  Navy,  2001).  Naval 
Expeditionary  Maneuver  Warfare  consists  of  military  operations  mounted  from  the  sea,  usually  on 
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short  notice.  They  are  carried  out  by  forward- deployed  or  rapidly  deployable,  self-sustaining  naval 
forces  tailored  to  achieve  a  clearly  stated  objective.  The  future  primary  platforms  for  Expeditionary 
Warfare,  known  as  the  “Amphibious  Assault  Triad”,  are  the  Landing  Craft,  Air  Cushion,  or  LCAC, 
the  Advanced  Amphibious  Assault  Vehicle  (AAAV),  and  the  Osprey  MV-22  tilt-rotor  aircraft. 
VIRTE  focused  on  these  diverse  vehicles  since  they  present  unique  simulation  and  training 
challenges,  as  well  as  a  component  contributed  by  the  United  States  Marine  Corps,  supporting  Close 
Quarters  Battle. 

Within  the  VIRTE  program,  a  range  of  simulations  are  currently  under  development.  Each  of  these 
systems  is  based  on  real-world  operational  requirements  and  each  is  intended  to  transition  to  real 
world  use  as  shown  in  Figure  1. 


V-  MWZZ 


CQB 

'(fUL 


Figure  1:  VIRTE  focuses  on  providing  deployed  training  solutions  to  the  Expeditionary  Warfare  community. 
This  community  includes  air,  land  and  sea  elements.  From  Far  Left  to  Right  Top  Row:  Landing  Craft  Air 
Cushioned  (LCAC);  Advanced  Amphibious  Assault  Vehicle  (AAAV)  The  MV-22  (Osprey)  is  a  tilt-rotor  craft 
currently  under  development;  Close  Quarters  Battle.  From  Far  Left  to  Right,  Bottom  Row:  Virtual  Environment 
(VE)  LCAC;  Virtual  Environment  (VE)  AAAV;  Virtual  Environment  (VE)  MV-22;  Virtual  Environment  (VE) 
CQB. 


Landing  Craft  Air  Cushioned  (LCAC) 

The  LCAC  is  the  only  member  of  the  triad  that  is  currently  fielded.  The  LCAC  is  a  high-speed,  over- 
the-beach  fully  amphibious  landing  craft  capable  of  carrying  a  60-75  ton  payload.  It  is  used  to 
transport  weapons  systems,  equipment,  cargo  and  personnel  from  ship  to  shore  and  across  the  beach. 
The  Navy  has  76  LCACs  in  service.  The  LCAC  crew  consists  of  three  positions,  the  Craftmaster 
(pilot),  the  Navigator,  and  the  Engineer  who  work  closely  together  to  operate  the  vehicle.  Currently, 
there  are  two  LCAC  Full  Mission  Trainers  (FMTs),  one  at  Assault  Craft  Unit  (ACU)  4  in  Dam  Neck, 
Virginia,  and  one  at  ACU  5  in  Camp  Pendleton,  California.  While  these  provide  excellent  high 
fidelity  training,  they  are  expensive  to  procure  and  operate.  The  LCAC  fleet  is  just  beginning  to  field 
a  Service  Life  Extension  Program  (SLEP)  which  completely  changes  the  operator  interface  to  the 
vehicle.  It  will  be  several  years  until  there  are  sufficient  SLEP  LCACs  fielded  to  transition  the  FMT 
to  the  SLEP  configuration. 


Advanced  Amphibious  Assault  Vehicle  ( AAAV) 

The  Advanced  Amphibious  Assault  Vehicle  (AAAV)  is  currently  in  the  prototype  stage  and  it  is 
scheduled  to  enter  Low  Rate  Initial  Production  in  FY  07.  The  AAAV  will  provide  the  capability  to 
move  a  combat  loaded  USMC  rifle  squad  at  over  20  knots  on  the  water  and  maneuver  cross  country 
with  the  speed  and  agility  of  the  Ml  tank.  It  will  replace  the  AAV7A1.  The  AAAV  crew  consists  of 
the  driver,  the  gunner,  and  the  vehicle  commander.  The  AAAV  is  in  the  System  Development  and 
Demonstration  (SDD)  phase  and  second  generation  prototypes  are  being  matured  and  prepared  for 
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production.  The  AAAV  will  be  supported  by  a  significant  investment  in  training  systems.  VIRTE  is 
focusing  on  transferring  technology  to  two  of  these,  the  schoolhouse  training  system  and  the  vehicle 
embedded  training  system. 

MV-22 

The  MV-22  Osprey  is  a  tilt-rotor  aircraft  that  will  provide  airlift  in  support  of  Expeditionary 
Maneuver  Warfare.  The  tilt-rotor  design  combines  the  speed,  range,  and  fuel  efficiency  normally 
associated  with  turboprop  aircraft  with  the  vertical  take-off/landing  and  hover  capabilities  of 
helicopters.  The  MV-22  crew  consists  of  a  pilot  and  co-pilot.  The  MV-22  is  currently  in  Low  Rate 
Initial  Production  (LRIP)  and  in  the  EiSMC  will  replace  the  CH-46E  and  CH-53D.  As  with  the 
AAAV,  there  is  a  significant  investment  in  training  systems.  Rather  than  try  to  insert  technology 
directly  in  the  MV-22  program,  we  are  concentrating  on  demonstrating  technologies  that  are 
applicable  to  all  aircraft  trainers. 

Close  Quarters  Battle  Military’  Operations  in  Urban  Terrain  (CQB  for  MOUT) 

CQB  for  MOUT  is  a  type  of  urban  warfare  that  is  distinct  from  the  commonly  portrayed  ‘jungle’  or 
‘desert’  combat  scenarios.  In  CQB  for  MOUT,  small  teams,  typically  in  groups  of  4,  are  tasked  with 
clearing  a  building  of  enemy  units,  room  by  room.  It  is  a  type  of  warfare  that  is  extremely  hazardous 
and  man-power  intensive.  Currently,  there  only  two  options  for  providing  CQB  for  MOUT  training. 
The  first  option  is  Live  training,  which  involves  moving  these  teams  through  mock-up  villages 
(‘combat  towns’)  using  simulated  ammunitions.  The  second  option  is  through  the  use  of  simple 
computer  generated  scenarios,  involving  the  projection  of  scenes  on  a  screen,  with  users  navigating 
through  the  scenario  using  simple  interfaces  such  as  joysticks  or  footpedals. 

Both  training  options  have  significant  disadvantages.  Live  training  has  significantly  reduced  realism 
due  to  the  numerous  safety  constraints  imposed  during  such  activities.  As  well,  this  type  of  training 
involves  the  use  of  pre-made  building  mock-ups  which  can  not  be  easily  reconfigured,  greatly 
reducing  the  range  of  potential  training  scenarios.  At  the  same  time,  the  simulations  currently  in  use 
also  fail  to  provide  a  strong  basis  for  performance  enhancement.  Current  systems  utilize  non-natural 
Human-Computer  Interfaces  and  therefore  cannot  teach  tactical  mobility  skills.  Lurther,  the  manner  in 
which  the  computer-generated  scenes  are  slaved  to  the  user’s  movement  does  not  support  the 
development  of  the  ‘move-look-shoot’  philosophy  crucial  to  this  form  of  combat.  Linally,  although 
CQB  for  MOUT  is  a  team-based  activity,  current  training  systems,  which  present  one  scene  to  an 
entire  team,  can  not  support  any  level  of  ‘individualized’  first  person  perspectives  that  is  part  and 
parcel  of  this  combat  environment. 

Prototype  Development 

While  the  real-world  analogues  of  each  VIRTE  component  serve  unique  roles  in  actual  combat  and 
impose  unique  training  requirement  upon  their  operators,  their  efforts  complement  each  other; 
standard  military  doctrine  relies  on  elements  from  each  to  achieve  the  overall  objective  (Department 
of  the  Navy,  1999).  In  parallel,  the  virtual  systems  are  designed  with  the  dual  purpose  of  supporting 
training  at  the  individual  level,  as  well  as  at  the  level  of  distributed,  team-based  events,  operating 
within  a  shared  synthetic  battlepsace. 

In  order  to  develop  a  suite  of  VE-based  training  tools  that  supported  each  of  these  Expeditionary 
Warfare  components,  within  the  Department  of  Defenses’  modeling  and  simulation  framework, 
several  parameters  had  to  be  defined.  First,  the  technologies  underlying  each  VE  system  must  be 
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compliant  with  the  US  Department  of  Defense  (DOD)  High  Level  Architecture  (HLA).  Second,  each 
of  the  simulations  had  to  share  the  same  virtual  battlespace.  Finally,  since  these  systems  were 
intended  to  transition  directly  to  the  users,  VIRTE  tried  to  minimize  program  license  costs  with  the 
vision  that  of  being  able  to  hand  out  CD  applications  to  anyone  that  wanted  them.  These  requirements 
led  to  the  selection  of  the  government  owned  JointSAF  as  the  simulation  environment.  OpenFlight 
was  selected  as  the  visualization  database  standard  and  allowed  each  of  the  development  teams  to 
choose  their  own  tools.  VIRTE’s  goals  in  this  domain  came  closest  to  the  vision  with  the  VELCAC 
which  uses  a  commercial  gaming  engine,  Gamebryo  (formerly  Netlmmerse).  VIRTE  Demonstrations 
II  and  III  will  all  use  Gamebryo. 

Early  prototypes  were  crucial  to  the  program.  In  a  two  year  span,  VIRTE  went  from  Demonstration  I 
concepts  to  deployable  prototypes.  VIRTE  developers  employed  a  Spiral  Development  process  and 
held  a  series  of  Intermediate  Feasibility  Experiments  (IFEs).  The  four  IFEs  were  events  where 
developers  deployed  their  latest  configurations  in  realistic  testing  environments  with  potential  users. 
All  of  the  VIRTE  simulations  are  PC  based  and  interoperable  in  the  same  virtual  battlespace.  They 
have  some  unique  differences  based  on  customer  requirements.  The  VEAAAV,  which  is  a  school 
house  training  prototype,  has  extensive  replication  of  the  physical  layout  of  the  crew  stations.  The 
VELCAC  uses  almost  all  virtual  displays,  with  the  exception  of  the  “throttle”  which  has  a  unique  feel 
and  gives  important  haptic  feedback  to  the  craftmaster.  The  VEHelo  is  the  only  VIRTE  Demo  I 
simulator  that  uses  a  HMD  in  the  deployed  configuration.  The  VEHelo  uses  a  Head  Mounted  Display 
(HMD)  which  combines  the  virtual  environment  and  live  video.  Each  of  these  approaches  has 
tradeoffs  and  these  are  brought  out  in  the  Training  Effectiveness  Evaluation. 

VIRTE  Demonstration  II,  Close  Quarters  Battle  for  Military  Operations  in  Urban  Terrain  (CQB  for 
MOUT)  has  a  much  greater  focus  on  “traditional”  Virtual  Environments.  The  most  unique  aspect  of 
Demo  II  is  that  it  puts  a  soldier  directly  into  the  virtual  environment  using  his  entire  body  and  weapon 
as  a  means  of  interaction.  Unlike  First  Person  Shooter  (FPS)  games,  which  use  keyboard  commands 
or  a  joystick,  VIRTE  is  prototyping  technology  which  allows  the  user  to  naturally  interact  with  the 
virtual  environment  either  walking  in  a  tracked  space  or  walking  in  place.  This  level  of  interaction 
with  the  Virtual  Environment  presents  exceptional  challenges  for  Human  Factors.  For  example,  what 
is  the  best  way  to  let  a  person  know  that  they  have  put  their  hand  through  a  virtual  wall?  How  do  we 
immerse  an  entire  fire  team  and  allow  them  to  have  natural  interactions?  In  Demo  II,  the  CQB 
training  system,  VIRTE  is  focusing  significant  investment  into  real  time  full  body  tracking,  collision 
detection,  distributed  audio  environments,  and  distributed  Virtual  Environments.  The  remainder  of 
this  paper  discusses  the  process  the  VIRTE  team  has  implemented  to  ensure  that  the  training  provided 
by  each  VE  system  provides  a  measurable  level  of  performance  enhancement  on  the  real  world  tasks 
they  are  designed  to  support. 

Training  Effectiveness  Evaluation  (TEE) 

In  order  to  better  understand  the  process  implemented  by  the  VIRTE  team  for  ensuring  training 
enhancement,  the  individual  components  comprising  this  user-centric  design  process  were 
categorized  as  elements  of  a  broader,  Training  Effectiveness  Evaluation  (TEE)  effort.  The  philosophy 
underlying  the  development  of  a  TEE  plan  includes: 

-  Up  Front  Analyses 

-  Provide  foundation  for  developing  systems 

-  Iterative  Integrated  Feasibility  Experiments 

-  Provide  iterative  feedback  for  system  developers 

-  Back  end  Analyses 
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-  Final  evaluation 

-  Lessons  Learned 

These  three  general  elements  can  be  operationalized  as  six  component  ‘building  blocks’  which  are: 

-  Task  Analysis 

-  Human  Computer  Interface  evaluation 

-  Team  Performance 

-  System  Usability 

-  VE  User  Considerations 

-  Training  Transfer 

The  manner  in  which  these  pieces  combine  is  somewhat  fluid.  In  a  broad  sense,  these  pieces  can  be 
grouped  into  three  general  phases.  Phase  one  consists  of  the  Task  Analysis  and  the  Human  Computer 
Interaction  evaluation.  The  first  element,  the  Task  Analysis  (TA),  seeks  to  identify  the  training 
objectives  and  the  scenario  elements  that  must  be  included  in  a  simulation  to  support  these  objectives 
as  well  as  providing  an  assessment  of  whether  currently  available  training  can  be  modified  so  that  a 
technology  solution  is  not  necessary.  For  TEE  purposes,  a  contextual  task  analysis  is  performed, 
which  focuses  on  the  behavioral  aspects  of  a  task  as  performed  in  a  given  operational  setting, 
resulting  in  an  understanding  of  the  general  structure  and  flow  of  task  activities  (Mayhew,  1999; 
Nielsen,  1993;  Wixon  &  Wilson,  1997).  The  second  element  Human-Computer  Interactions  (HCI) 
focuses  on  identifying  the  requirements  for  sensory  modality  integration,  as  well  as  evaluating  current 
hardware/software  technologies  supporting  these  interactions,  and  providing  guidance  for  integrating 
them  into  a  simulation.  Once  these  efforts  are  completed,  the  information  is  then  passed  on  to  the 
Simulation  Developers,  who  may  then  commence  their  efforts. 

The  second  phase  involves  an  iterative  evaluation  process,  in  which  the  system  is  evaluated  at  key 
points  within  the  development  lifecycle.  These  points,  termed  Integrated  Feasibility  Experiments  are 
points  at  which  the  TEE  team  can  evaluate  progress  along  the  dimensions  determined  in  phase  1. 
There  are  two  key  portions  to  this  phase.  The  first  is  System  Usability,  which  evaluates  how 
accommodating  the  overall  VE  design  is  for  use  by  the  layperson  (Stanney,  Mollaghasemi,  &  Reeves, 
2000)  and  provides  redesign  recommendations  to  the  developers.  The  second  is  VE  User 
Considerations,  which  addresses  both  evaluations  of  the  side  effects  encountered  during  exposure  to 
VE,  as  well  as  aftereffects  arising  following  this  exposure. 

The  final  phase  focuses  on  performing  a  system-wide  evaluation,  and  then  documenting  these 
findings  in  a  ‘lessons  learned’  format.  Since  all  the  training  systems  under  development  for  VIRTE 
focus  on  crew  or  team-based  application,  the  first  component  of  this  effort  is  Team  Performance, 
which  addresses  how  well  a  simulation  supports  team-based  activities  (Cannon-Bowers,  &  Salas, 
1998).  Perhaps  the  most  important  element  of  TEE,  which  validates  all  the  efforts  described  thus  far, 
is  an  evaluation  of  the  degree  to  which  Training  Transfers  from  the  VE  to  the  real  world  scenario 
(Carretta  &  Dunlap,  1998;  Lathan,  Tracey,  Sebrechts,  Clawson,  &  Higgins,  2002;  Waller,  Hunt,  & 
Knapp,  1998).  Figure  2  provides  an  overview  of  how  these  components  can  be  integrated  into  a 
comprehensive  TEE  plan. 
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Figure  2:  Schematic  of  elements  comprising  the  VIRTE  Training  Effectiveness  Evaluation,  and  the 
relationship/flow  structure  linking  them. 

As  Figure  2  suggests,  TEE  is  not  simply  a  process  that  is  performed  once,  at  any  single  stage  in  the 
design  of  a  VE  simulation.  Rather,  as  with  traditional  usability  engineering  practices  (Nielsen,  1993), 
it  must  be  interwoven  throughout  the  entire  design  process.  Moreover,  each  of  the  proposed  TEE 
elements  is  intertwined  with  the  others.  Ultimately,  the  six  aspects  of  TEE  should  combine  to  form  a 
seamless,  continuous,  evaluation  of  the  overall  VE  training  simulation,  leading  to  a  well  thought  out, 
well-designed  system  that  will  enhance  training  while  reducing  the  cost  of  providing  this  training. 
The  research  objectives  of  VIRTE  were  structured  so  that,  while  each  VE  platform  utilized  aspects  of 
each  TEE  component,  each  platform  was  also  singled  out  to  highlight,  from  the  science  and 
technology  perspective,  issues  and  applications  within  these  elements. 

Task  Analysis 

One  of  the  most  basic  challenges  that  any  VE  developer  faces,  prior  to  designing  a  new  VE 
simulation  is  How  to  design  a  system  that  will  support  the  users  ’  training  needs?  The  commonly 
accepted  approach  for  answering  this  question  is  to  conduct  a  task  analysis.  While  there  are  many 
types  of  task  analysis  (Chapman,  Schrage,  &  Smith,  2001),  it  is  possible  to  select  one  basic 
methodology  by  understanding  the  characteristics  of  the  user  community.  VIRTE  is  concerned  with 
providing  experts  with  training  on  abstract  concepts  rather  than  with  training  novices  to  perform 
specific  perceptuomotor  tasks,  hence  a  cognitive  task  analysis  (CTA). 

Through  this  effort,  training  objectives  and  evaluation  metrics  were  developed  for  each  platform. 
Considering  the  training  of  the  LCC  community,  it  was  determined  early  on  that  the  VELCAC 
system’s  primary  use  would  be  as  a  teaching  supplement  to  acquaint  highly  skilled  crews  with  a  new 
cockpit  instrumentation  layout  and  corresponding  upgraded  features  (the  Service  Life  Extension 
Program,  or  SLEP,  LCAC  configuration).  The  resultant  CTA  identified  those  features  of  VELCAC 
that  would  best  complement  the  SLEP  differences  course,  a  two-week  training  course  designed  to 
acquaint  crewmembers  with  the  new  LCAC  features.  The  key  training  objective  for  the  VE  AAAV 
was  identified  as  being  the  provision  of  a  gunnery  training  system.  Hence,  the  task  analysis  focused 
on  techniques  for  providing  real  time  and  after  action  feedback  to  enhance  speed  and  accuracy  of  this 


RTO-MP-HFM-101:  Paper  2 


Page  7  of  12 


ORGANIZATION 


task.  The  focus  of  the  VE  Helo  system  was  on  providing  enhanced  training  for  visualizing  spatial 
relationships  in  preparation  for  low  level,  Nap  of  Earth  (NOE)  flight.  Consequently,  the  VE  Helo  task 
analysis  focused  on  identifying  the  cognitive  components  of  route  learning  and  navigation.  Finally, 
the  intended  use  of  the  CQB  training  system  is  to  provide  a  means  for  maintaining  skill  proficiency 
during  extended  deployments,  as  well  to  provide  an  interactive,  three-dimensional  mission  rehearsal 
system.  The  CQB  task  analysis  focused  not  only  on  core  skills  and  cognitive  components  of  basic 
CQB  maneuvers,  but  also  on  understanding  tactics  and  strategies  that  the  Opposing  Forces  (OpFor) 
might  use. 

Human  Computer  Interaction 

The  overall  goal  of  the  HCI  evaluation  is  to  identify,  based  on  the  CTA,  which  modalities  (haptics, 
visual,  sound,  etc.)  must  be  represented  within  the  VE  to  in  order  to  provide  an  effective  training 
experience.  Since  the  bulk  of  the  VIRTE  systems  are  vehicle-  based,  the  two  primary  senses  under 
consideration  are  vision  and  haptics.  Although  much  work  has  been  done  exploring  the  visual 
modality,  the  integration  of  haptics  within  a  VE  simulation  represents  a  unique  challenge  because, 
unlike  vision  (and  audition),  the  haptic  system  provides  both  sensation/perception  and  the  means  to 
manipulate. 

The  bulk  of  VIRTE’s  HCI  effort  focused  on  the  VELCAC  system.  Several  sources  of  information 
were  examined  during  this  evaluation.  These  included  a  requirements  document,  user  profiles,  results 
from  user  interviews,  the  TA,  field  observations,  and  knowledge  of  the  operational  environment.  In 
addition,  a  literature  review  examining  performance  issues  relating  to  visual  displays  and  haptic 
systems  was  conducted.  The  analysis  of  this  information  provided  a  set  of  criteria  by  which  decisions 
regarding  visual  and  haptic  displays  could  be  made.  Critical  information  derived  from  these  sources 
suggested  that  of  the  three  crew  positions  under  consideration,  only  the  Craftmaster  (the  crewmember 
who  flies  the  LCAC)  would  need  physical  controls  (yoke,  foot  pedals,  and  throttle)  in  the  VELCAC 
and  a  wide  field  of  view,  in  order  to  maintain  adequate  situational  awareness  while  piloting  the 
(virtual)  craft.  The  other  two  crewmembers  (the  Navigator,  who  monitors  instant  position,  and  the 
Engineer,  who  monitors  LCAC  system  function)  spend  the  majority  of  their  time  observing  and 
interacting  with  touch  screen  interfaces  on  the  instrument  console.  Consequently,  the  final  version  of 
VELCAC  utilized  a  panoramic-like,  flat-screen  based  visual  display.  The  Craftmaster  position  was 
provided  with  the  actual  controls  interfaces,  while  the  other  two  positions  were  provided  with  an 
interface  to  their  instruments  based  on  using  a  mouse  to  point  and  select  specific  options. 

Usability 

System  usability  focuses  on  evaluating  how  accommodating,  intuitive  and  easy-to-use  software  and 
hardware  are  for  use  by  the  layperson.  The  assessment  of  a  system’s  usability  typically  involves  an 
initial  expert  evaluation  of  the  system  to  identify  any  design  weaknesses  in  the  prototype  system, 
which  includes  an  observational  assessment  of  users  interacting  with  the  VE,  an  analysis  of  these  data 
to  identify  which  features  are  less-than  optimal  and  why  and,  the  development  of  a  set  of  ranked  and 
ordered  redesign  recommendations  for  system  developers  to  follow.  As  should  be  apparent  Usability 
evaluation  is  most  useful  when  it  is  incorporated  as  part  and  parcel  of  an  iterative  design  and 
development  process. 

As  an  example  of  how  this  iterative  process  works,  during  each  IFE  the  usability  assessment  of  the 
VELCAC  began  with  an  expert  evaluation  of  the  current  version  of  VELCAC,  conducted  by  two 
usability  engineers.  This  assessment  was  then  followed  by  user  observation,  by  these  same  experts,  of 
LCAC  crewmembers  interacting  with  various  SLEP  VELCAC  stations.  The  results  of  these  efforts 
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were  used  to  provide  redesign  recommendations,  as  well  as  to  establish/validate  usability  metrics  and 
their  associated  acceptability  criteria  for  future  usability  testing  of  SLEP  VELCAC  as  more  mature 
iterations  of  the  system  are  developed. 

VE  User  Considerations 

Typically,  the  impact  of  Virtual  Environments  on  users  has  been  typified  in  terms  of  a  collective 
group  of  symptoms  known  as  Cybersickness.  Operationally,  the  majority  of  the  research  into  these 
symptoms  has  focused  on  acute  symptoms,  such  as  eyestrain  and  blurred  vision,  rather  than  on  a  more 
holistic  approach,  which  would  focus  on  impact  on  warfighter  readiness.  Yet,  as  the  military 
continues  its  push  towards  using  VE  systems  to  provide  just  in  time  training  (Cohn,  Muth, 
Schmorrow,  Brendley  &  Hillson,  2002),  it  is  becoming  increasingly  evident  that  this  level  of  research 
will  be  the  one  most  critical  for  evaluating  the  utility  of  using  VE  systems.  Currently,  the  military  has 
restrictions  delineating  the  amount  of  time  an  individual  must  wait  between  exposure  to  simulated 
training  and  participating  in  the  actual  real  world  effort. 

For  the  purposes  of  VIRTE,  whose  simulators  target  Naval  forces  and  may  ultimately  be  placed 
aboard  ship  (Cohn  et  al,  2002),  a  critical  concern  is  the  sensory  decoupling  that  arises  between  the 
visually  indicated  motion  reference  frame,  provided  through  immersion  within  the  VE,  and  the 
physically  indicated  motion  reference  frame,  provided  through  the  (moving)  platform  of  the  ship. 
Such  sensory  discordance  may  be  particularly  troublesome,  both  during  training  as  well  as  following 
training.  Preliminary  results  (Cohn  et  al.,  2002)  suggest  that  in  this  situation  side  effects,  such  as 
motion  sickness  will  result.  VIRTE  is  actively  pursing  two  solutions  to  this  sensory  discordance 
challenge.  The  first  focuses  on  developing  technology  to  re-introduce  the  coupling  between 
physically  sensed  and  visually  indicated  motion.  The  physically  indicated  motion  of  the  ship  is  locked 
to  the  motion  of  the  entire  framework  of  the  virtual  scene  being  displayed  through  the  VE  in  such  a 
manner  that  a  trainee  experiences  both  the  intended,  virtual  motion,  as  well  as  a  degree  of  motion 
recoupling.  Preliminary  results  suggest  that  this  approach  holds  promise  (Brendley,  Marti,  Cohn  & 
DiZio,  2002).  At  the  same  time,  it  is  crucial  to  quantify  the  impact  these  decouplings  have  on 
warfighter  readiness.  Consequently,  VIRTE  is  developing  a  Warfighter  Readiness  Toolkit  that 
combines  cognitive,  perceptual  and  physiological  evaluation  tools  that  can  detect  subtle  changes  that 
might  lead  to  deleterious  side  effects  that  could  impact  warfighter  readiness. 

Team  Performance  and  Training  Transfer 

Ultimately,  each  of  the  previously  mentioned  efforts  are  useful  only  to  the  extent  that  they  are  able  to 
support  the  primary  goal  of  developing  any  VE  simulation,  namely,  to  provide  a  level  of  training  that 
translates  to  enhanced  performance  of  the  real  world  task  being  simulated  (Lathan  et  al.,  2002). 
Despite  the  fact  that  VE  systems  continue  to  be  pushed  into  the  training  application  domain,  there  are 
few  convincing  illustrations  of  the  transfer  of  complex  skills  from  VE  training  to  their  corresponding 
real-world  applications  (Cohn,  Helmick,  Meyers  &  Bums,  2000).  A  critical  limiting  factor  in  most 
such  efforts  is  that  they  fail  to  utilize  adequate  performance  metrics,  relying  instead  on  users’ 
anecdotal  evaluations  or  perceptions  of  training  utility  rather  than  statistically  determined  measures. 
Thus  most  results  are  difficult  to  interpret  both  in  terms  of  the  underlying  processes  that  lead  to  either 
positive  or  negative  transfer,  as  well  as  in  terms  of  the  meaning  of  outcome. 

The  most  common  approach  for  assessing  training  transfer  is  to  compare  performance  between  two 
groups  of  trainees,  an  experimental  group  that  receives  simulator  training  and  a  control  group  that 
receives  all  of  its  training  in  the  real  world,  along  some  predefined  set  of  metrics  (Boldovici,  1987; 
Cohn  et  al.,  2000;  Murdock,  1957).  It  is  important  to  note  that  while  it  may  be  possible,  in  a  general 
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sense,  to  develop  standard  approaches  for  this  effort,  ultimately,  the  metrics  thus  developed  will  be 
context  specific.  Thus,  for  each  type  of  VE  system,  it  is  critical  to  identify  differences  in  process  and 
outcome  behaviors  between  individuals  who  receive  VE  training  and  those  who  do  not  receive  that 
training  when  performing  operational  tasks  in  the  real  world  environment. 

Importantly,  for  purposes  of  a  transfer  study,  each  of  the  real  world  platforms  being  simulated 
through  VIRTE  involves  interactions  between  team  members.  Consequently,  measures  of  changes  in 
team  processes  (e.g.,  exchange  of  information,  backup  behaviors  during  times  of  high  workload) 
serve  as  a  critical  measure  of  training  effective  and  directly  impact  the  degree  to  which  training  will 
transfer.  At  the  same  time,  more  task  specific,  outcome  measures  should  also  improve,  such  as 
improved  timing  (e.g.,  at  checkpoints  and  way-points)  and  accuracy  (e.g.,  less  discrepancy  between 
actual  and  planned  speeds  and  course),  which  should  occur  if  there  is  positive  transfer  of  training 
from  the  VE  to  the  real  world. 

Summary 

While  many  elements  of  Virtual  Environment  development  are  platform  specific,  it  is  possible  to 
distill  a  set  of  principles,  which  if  followed,  should  translate  to  a  more  effective  training  system. 
These  principles  can  be  implemented  early  on,  as  part  of  the  task  analysis  and  human  computer 
interaction  evaluation  efforts  and  can  continue,  throughout  the  development  lifecycle  as  a  usability 
and  user  consideration  assessment.  Ultimately,  the  success  of  these  efforts  will  be  established  through 
a  comprehensive  training  transfer  study. 
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